The present section is intended to introduce the reader to various aspects of art, which may be related to various aspects of the present invention that are described and/or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present invention. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.
Image acquisition devices project a three-dimensional scene onto a two-dimensional sensor. During operation, a conventional capture device captures a two-dimensional (2-D) image of the scene representing an amount of light that reaches a photosensor (or photodetector or photosite) within the device. However, this 2-D image contains no information about the directional distribution of the light rays that reach the photosensor (which may be referred to as the light-field).
Moreover, it is more and more frequent to post-process the image data captured by the sensor, and to run computational photography algorithms on the acquired signals.
However, in order for such image data processing to be performed correctly, it is necessary to have accurate calibration data relating to the image acquisition device used to capture such image or video data.
Notably, when considering a sensor device observing an object space from the image space of an optical system, it is necessary to estimate, for each pixel of the sensor, to which direction(s), or beam(s), it corresponds in the object space (i.e. which portion of the object space is sensed by this pixel). In the present disclosure, the terms “object space” and “image space” respectively stand for the input and output optical spaces usually defined in the optical design discipline. Hence, the “object space” is the observed scene in front of the main lens of an image acquisition device, while the “image space” is the optical space after the optical system of the image acquisition device (main lens, microlenses, . . . ) where the imaging photosensor captures an image.
Among the required calibration data, what is first needed is to identify the chief ray direction in the object space of the beam corresponding to a sensor pixel. A second need is to know the (angular) shape of the geometric beam surrounding the chief ray, and sensed by the pixel.
Such a need is well known in the art. In “Spatio-Angular Resolution Tradeoffs in Integral Photography”, Rendering Techniques 2006 (2006): 263-272, Todor Georgeiv et al. express it as follows: “in order to reconstruct the true irradiance corresponding to the illuminated part of each pixel we would need to know exactly what percentage of it has been illuminated, and correct for that in software. In other words, we would need very precise calibration of all pixels in the camera. However, captured pixel values are affected by tiny misalignments: A misalignment of a micrometer can change a boundary pixel value by more than 10%. This problem gets very visible when the lenslets get smaller.”
According to known prior art techniques, optical systems calibration mainly use checkerboards or grids of points in the object space to estimate the position of corners or intersection points on the acquired images in the image space. For a given optical configuration (a given zoom/focus of the optical acquisition device), grid points or corners positions are estimated with sub-pixel image processing techniques, and these estimates are provided to a model generalizing the estimated positions to the entire field of view.
Such a perspective projection model is usually taken as a starting point for optical acquisition devices calibration. It is then supplemented with distortion terms, in order to get very precise calibration of all pixels in the camera.
In “A Generic Camera Model and Calibration Method for Conventional, Wide-Angle, and Fish-Eye Lenses”, Pattern Analysis and Machine Intelligence, IEEE Transactions on 28, no. 8 (2006): 1335-1340, Kannala et al. consider that the perspective projection model is not suitable for fish-eye lenses, and suggest to use a more flexible radially symmetric projection model. This calibration method for fish-eye lenses requires that the camera observe a planar calibration pattern.
In “Multi-media Projector-Single Camera Photogrammetric System For Fast 3d Reconstruction”, International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, Commission V Symposium, pp. 343-347. 2010, V. A. Knyaz proposes the use of a multimedia projector to simultaneously calibrate several cameras in a 3D reconstruction context.
Existing calibration methods hence rely on a global model transforming the geometry in the object space to the geometry in the image space. However, such prior art techniques are not suited for light field acquisition devices, which show a complex design and embed optical elements like lenslet arrays, which do not always follow specifications with all the required precision.
It is actually recalled that light-field capture devices (also referred to as “light-field data acquisition devices”) have been designed to measure a four-dimensional (4D) light-field of the scene by capturing the light directions from different viewpoints of that scene. Thus, by measuring the amount of light traveling along each beam of light that intersects the photosensor, these devices can capture additional optical information (information about the directional distribution of the bundle of light rays) for providing new imaging applications by post-processing. The information acquired/obtained by a light-field capture device is referred to as the light-field data.
Light-field capture devices are defined herein as any devices that are capable of capturing light-field data. There are several types of light-field capture devices, among which:                plenoptic devices, which use a microlens array placed between the image sensor and the main lens, as described in document US 2013/0222633;        a camera array, as described by Wilburn et al. in “High performance imaging using large camera arrays.” ACM Transactions on Graphics (TOG) 24, no. 3 (2005): 765-776 and in patent document U.S. Pat. No. 8,514,491 B2.        
For light field acquisition devices, a precise model of the optics (including defects such as microlens array deformations or misalignment) is more complex than with classical single pupil optical systems. Moreover, with light field acquisition devices, blur or vignetting can affect image forming, distorting the relationship between a source point and its image on the sensor. Last, the notion of stationary Point Spread Function, which is used when calibrating conventional image acquisition devices, does not hold for light field acquisition devices.
It is hence necessary to provide calibration techniques, which are suited for calibrating light field acquisition devices.
Specific calibration methods and models have hence been proposed for plenoptic or camera arrays acquisition, as in “Using Plane+Parallax for Calibrating Dense Camera Arrays”, Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, vol. 1, pp. 1-2. IEEE, 2004 by Vaish et al. This document describes a procedure to calibrate camera arrays used to capture light fields using a planar+parallax framework.
However, such a technique does not allow determining with enough precision the chief ray position and direction estimation in the object space, nor the geometrical characteristics of the object space region seen by a given pixel. Such an object space region may also be called a pixel beam in the present disclosure.
It would hence be desirable to provide a new technique for calibrating image acquisition devices, which would allow calibrating light field acquisition devices, and which would not show the drawbacks of the prior art techniques. More precisely, it would be desirable to provide a new technique for calibrating image acquisition devices, which would allow estimating accurately characteristics of a pixel beam directed to a given pixel on a sensor of the image acquisition device.